Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FSRS: ignore initial reviews until first longer break #3734

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

BtbN
Copy link

@BtbN BtbN commented Jan 17, 2025

I'm dealing with a deck where I practically don't know any of the new cards that show up (It's a Kanji-Learning-Deck).
So that results in practically all first ratings to be "Again", which results in relatively poor performance of FSRS for that deck for me.
It consistently thinks I'm much better than I actually am, resulting in lower retention than the configured desired value.

I also initially did some mistakes in my learning process(while still using SM2), and hit "Good" as initial rating on cards I learned on the Sibling Card minutes prior, which should actually have been "Again".

I have however consistently had a 3h learning step, so I got the idea that this patch implements:
Ignore all ratings until after that first 3h break. That way FSRS won't see just a wall of Again (and my faulty early Good-Reviews), but actually a pretty well balanced mix, with Good being the by far most common, but also quite a few Hard and some Again ratings.

Here's the resulting parameters of optimizing that deck, once with a cutoff of 1.1.2024, and once the full collection with all the broken early reviews:

with patch(2024+): 0.9740, 1.3670, 5.4289, 13.6145, 6.7360, 0.9580, 3.7119, 0.0010, 1.4519, 0.4317, 1.0705, 2.0221, 0.0013, 0.3570, 2.4720, 0.0738, 2.9898, 0.2773, 0.6563
with patch(full): 1.9143, 2.3422, 6.2829, 12.1299, 5.9645, 0.8355, 3.9036, 0.0010, 1.3884, 0.2687, 1.0849, 2.3507, 0.0036, 0.3997, 3.0904, 0.0083, 5.9993, 0.1425, 0.0225
no patch(2024+): 2.2672, 13.6816, 16.2068, 18.1442, 5.8833, 0.8862, 3.6219, 0.0010, 1.5006, 0.3336, 1.0599, 2.1953, 0.0113, 0.3936, 2.6687, 0.0007, 2.9898, 0.2133, 0.2458
no patch(full): 2.8489, 9.1171, 19.3318, 31.9068, 5.7983, 0.8792, 3.7462, 0.0013, 1.3907, 0.1841, 1.0353, 2.4633, 0.0029, 0.3701, 3.1397, 0.0000, 6.0000, 0.1994, 0.1922

As demonstrated by the initial stabilities, this has pretty much solved the issue of a way too high stability for Cards initially rated "Good", which mostly affects ~1000 cards from my early reviews.

But even without such a mistake in a lot of review histories, I think this approach is worthwhile.
Specially when learning completely new material via a deck, all initial ratings will be Again. The first rating after the first break will almost certainly be more varied.

Obviously the code is not ready like this. It at the very least needs to be configurable via an option. Just looking for opinions here.
I also found no obvious way how to access the deck setting from that place. Looks like it'll have to be passed on as a parameter for quite a while.

@brishtibheja
Copy link
Contributor

In my opinion, we should strictly avoid adding more options to deck options. Instead, is there a way you can do this via custom scheduling like this one: https://kuroahna.github.io/anki_srs_kai/index.html

There is already too many options and I worry many users find this too complicated. What do you think?

@BtbN
Copy link
Author

BtbN commented Jan 17, 2025

This one could comfortably be hidden in the advanced options, since it's for a pretty specific case.
I don't see an easy way to implement this via custom scheduling without completely re-implementing FSRS there.

@Expertium
Copy link
Contributor

Agreed. We need to move in the direction of simplifying and streamlining FSRS, not making it even more complicated (it's already way above the complexity level that an average person can comfortably deal with).

@YukiNagat0
Copy link
Contributor

Sorry, but I think, that this is a very bad idea: many users have their own style of reviewing. Also, a lot of people in Anki community (and even docs, I believe) recommend to use Again as first mark, when you learning a completely new to you card (piece of information).

It is absolutely, no-good decision to cut initial reviews on some arbitrary duration.

I strongly believe, that FSRS handles always Again mark on new card correctly. (It is exactly my situation, where I always press Again on every new card, and I very satisfied with FSRS scheduling)


My suggestion for your case is to manually tweak FSRS weights (particularly, the [0] initial stability (Again)) to match your desired behavior. I recommend you to use https://open-spaced-repetition.github.io/anki_fsrs_visualizer/ for this.

@BtbN
Copy link
Author

BtbN commented Jan 18, 2025

That's what I've been doing so far, but it's not working out.
Also largely cause I have so many cards where I handled the initial learning phase very differently.
Retention is just getting worse and worse, just tweaking FSRS parameters isn't working.
There seems to be some more fundamental issue, so this is my attempt at providing FSRS with a more consistent and more varied starting point to work with, which allows it to safely Optimize the whole deck.

Cause that's another issue. Right now I have to set the Optimizer to ignore nearly the first 8 months or so of the deck, cause it goes completely bonkers if it sees the inconsistent reviews there.
And since that means it never sees a lot of the cards I struggle with(setting a cutoff point makes it ignore all those cards entirely, not just their old reviews), it might not optimize correctly. I don't think I have an easy way to verify this, but at least it sounds somewhat plausible to me.
With this Approach, I can take all reviews into account with minimal issues, so the Optimizer can look at the reviews later reviews of old cards fine.

@Luc-Mcgrady
Copy link
Contributor

Luc-Mcgrady commented Jan 18, 2025

Have you tried optimising on the beta? Recency weighting may help with your issue.
open-spaced-repetition/fsrs-rs#260

@BtbN
Copy link
Author

BtbN commented Jan 18, 2025

All the parameter-examples above are from latest main, which right now is identical with the 25.01 beta.
So it still goes bonkers with the initial stability of cards rated Hard/Good.

From how I understand it, the recency-weighting does not affect the initial stability whatsoever, the very first rating still gets special treatment.

@iamllama
Copy link
Contributor

this has pretty much solved the issue of a way too high stability for Cards initially rated "Good", which mostly affects ~1000 cards from my early reviews.

If you've only just recently changed your reviewing habit (i.e using Again instead of Good for new cards), you might be better off ignoring reviews before that date

once with a cutoff of 1.1.2024

Is 1.1.2024 when you changed your reviewing habit or an arbitrary date that still includes faulty reviews?

And since that means it never sees a lot of the cards I struggle with(setting a cutoff point makes it ignore all those cards entirely, not just their old reviews), it might not optimize correctly.

But your reviews for those cards are faulty as you've said. And with this change you're assuming that everyone's initial reviews are faulty as well.

@user1823
Copy link
Contributor

user1823 commented Jan 18, 2025

I don't think that this change should be implemented in Anki for the same reasons that have been mentioned by others above.

I also initially did some mistakes in my learning process(while still using SM2), and hit "Good" as initial rating on cards I learned on the Sibling Card minutes prior, which should actually have been "Again".

For solving your case, there are three options:

  • move the problematic cards into their own deck with a separate preset.
  • edit the database to change the first ratings of the problematic cards from Good to Again (and any other inconsistencies, which you would know better than me)
  • edit the database to delete the ratings before the first 3 hour interval (basically the same change as you are trying to do with this PR, but permanent and only in your collection).

By the way, I don't think that pressing Good as the initial rating on some cards would cause them to have such high initial stability just because most of the other ratings are Again. There should be a large amount of data showing that high stabilities would fit the data better for the optimizer to give such high values.

You can get a summary of how well you recall a newly introduced card for different first ratings from the Python optimizer. It generates a file called something like S0_dataset.tsv

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants